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ABSTRACT 


Through the increasingly common use of devices that provide ubiquitous sensor data 
such as wearables, mobile phones, and Internet-connected devices of the sort, privacy 
challenges are becoming even more significant. One major challenge that requires more 
focus is bystanders’ privacy, as there are too few solutions that solve the issue. Of the 
solutions available, many of them do not give bystanders a choice in how their private 
data is used. Bystanders’ privacy has become an afterthought when it comes to data 
capture in the forms of photographs, videos, voice recordings, etc. and continues to 
remain that way. This thesis provides a solution to enhance bystanders’ facial privacy by 
developing a wearable device called FacePET that provides a way for bystanders to 
protect their privacy and give consent. FacePET was evaluated using experiments to 
detect faces in photos when users wore the device and by performing a usability study 
with 21 participants. We found that FacePET was successfully able to block 15 of the 21 
participants’ faces, yielding a success percentage of 71%. We found through the 
usability study that a majority of the participants would be willing to use FacePET, ora 
similar device, daily for their facial privacy protection. 


Keywords: Bystanders’ privacy; Face detection; Face recognition; Privacy; Wearables; 


Internet of Things. 
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Chapter 1. Introduction 


1.1 Bystanders’ Privacy 


According to Ericsson’s Mobility Report [1], there are more than four billion 
smartphones subscriptions in the world. The availability of these devices with high- 
resolution cameras, mobile Internet connectivity, and the development of artificial 
intelligence techniques such as deep learning can expose individuals to privacy issues. 
Among these issues is bystanders’ privacy [2 — 3] which is the issue that arises when a 
device collects sensor data (such as photos, sound or video) that can be used to identify 
bystanders who may have not given consent for them to be identified. It is worthy to 
note that this issue arises with any camera-enabled Internet of Things (loT) device such 


as web/security cameras and drones. 


As an example in which bystanders were identified by using photos of their faces 
without consent, in 2016 a Russian photographer took photos of bystanders at a subway 
station and was able to identify them using free software available on the Internet [4]. 
The bystanders later knew about their identification through news reports. Examples 
like this one underscore the risks that people are exposed to with respect to their facial 


privacy given the technology currently available. 


Looking at it from a human-computer interaction standpoint, research in the early 2000s 
found that cellphone use in public spaces was offensive to some people [5] seeing as 
they presented a conflict of social spaces where the user occupied both the physical and 
virtual spaces at the same time. With wearable devices in today’s world such as smart 
glasses also including cameras and microphones, strong privacy concerns are being 


provoked by the collection and sharing of data over the Internet without permission, 


thereby directly threatening bystanders’ space and autonomy [6]. 








1.2 Problem Statement 


With such a rise in concerns about bystanders’ privacy from consumers, there is yet to 
be a viable solution that allows for bystanders to be more in control. Most research in 
the past decade or so have been more focused on the privacy of the wearer instead of 
whoever else’s privacy can be affected by the data collection effort. There are several 
reasons why this issue needs more attention from researchers and the general 
consumers. For one, consumers lack the means to control their privacy when using 
wearable devices. Another reason is that bystanders do not want their privacy to be 
exposed when somebody is using a wearable device nearby. Lastly, no standard 
approach exists to handle third-parties in consumer wearables. Thus, researchers began 


developing ways to combat bystander’s privacy by various means. 
1.3 Our Contribution 
We summarize our contributions as follows: 


e The design and implementation of a wearable device, called FacePET, that uses 
LED lights to block a camera’s ability to detect faces. The device is geared 
towards preserving the privacy of whomever is to wear it. 

e Aconsent protocol over Bluetooth that provides users wearing the FacePET a 
Way to give consent. 

e Auser study on wearable, Internet of Things devices geared towards facial 


privacy protection. 
Thesis Organization 


The first chapter of this thesis includes an introduction of what bystanders’ privacy is, 
the problem statement, and what our contribution is to the area of study. The second 
chapter consists of a general overview of face detection and recognition algorithms, the 


methods of bystanders’ privacy systems, the design issues of those systems, recent 


protection methods developed by other researchers, and an evaluation of how well the 








methods perform. Chapter three provides a description of the wearable system’s 
development which includes detailed explanations of the system’s components as well 
as the roles of each built application and how they work. Chapter four analyzes and 

discusses the results of the tests done with human participants using the device. Lastly, 


chapter five concludes this thesis’s research and considers recommendations for future 


work. 








Chapter 2. Background 


2.1 Introduction 


Before getting into our own solution regarding the issue of bystanders’ privacy, we must 
first study the research and solutions others have done that have helped us get to the 
point we are at now. As we go through this chapter, we will analyze and explain exactly 
how face detection and recognition work as well as the algorithms that make them 
possible. We will also present a taxonomy of bystanders’ facial privacy solutions, and a 
review of current methods available in the literature to enhance the facial privacy of 
bystanders. As a note, the information in this chapter, as well as in Chapter 3, has been 


published in the Electronics journal [41]. 
2.2 Face Detection and Recognition 


Even though research in face detection and recognition dates back from the 1970’s [7 — 
8], the advent of imaging sensors embedded in smartphones and digital cameras in 
conjunction with social networks have made research in the development of these 
algorithms to flourish in the last decade. Private companies (e.g., Facebook [9]) in 
addition to law enforcement agencies [10 — 11] are using algorithms to detect faces for 
business and law enforcement purposes. In computer vision and image processing, face 
detection is the problem of detecting if a face is present in a photo/video and face 


recognition is the problem of associating a face in a photo/video with an identity. 


The processes involved in the detection and recognition of faces in photos and/or video 
recordings are presented in Figure 1. Initially photos or videos are captured using some 
type of digital camera embedded in an loT device such as a mobile phone, a drone, or 


Internet-connected camera (image capture phase). Then, these digital photos/videos 


are passed through software that checks if faces are present in the photo/video (face 





detection phase). Finally, if faces are detected, then the face recognition phase is 


performed. The output of this last phase are the identities of the detected faces. 
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Figure 1. Processes for face detection and recognition. 


The development of fast and practical implementations of face detection algorithms in 
portable devices was possible through the work of Viola — Jones who developed a face 
detector that became a standard technique for this task [12]. Viola — Jones’ work is 
based on three main ideas [13]: (1) the utilization of an image representation (a data 
structure called “integral image”) that facilitates the extraction of simple features 
(called “Haar-like features”); (2) the utilization of a simple and efficient classifier based 
on the AdaBoost machine learning algorithm to select the most promising features to 
detect faces; and (3) the utilization of a combination of classifiers organized in sequence 
(called “cascade classifiers”) which allows to quickly discard regions of the image while 
concentrating on the most promising regions where faces may lie [13]. In the algorithm, 


a Haar-like feature is calculated as follows [14]: 
h(r1, r2) =s(ri) — s(r2) 


where s(r1) is the average of the intensities of the pixels in the “white” regions, and 
s(r2) is the average of the pixel intensities in the “black” regions as specified by patterns 


defined by a Haar-like feature. In their paper, Viola — Jones use the basic Haar-like 


features shown in Figure 2. 
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Figure 2. Haar-like features in the Viola and Jones face detection algorithm [13]. 
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Figure 3. Cascade classifiers in Viola — Jones [12]. 


The goal on the use of these features is to guide the face detection algorithm to find 
better regions of interest in which a face may possibly lie. Before this algorithm was 
developed, other algorithms already did face detection, but they relied on techniques 
using pixel positions and relations between pixels in an image, with more expensive 


computational cost than the Viola — Jones’ approach [12]. 


The Viola — Jones algorithm calculates the values of these Haar-like features by making 
use of windows (subregions) with different sizes from the original image. Once the 
features are calculated for all windows, the windows are passed through a classifier that 
outputs “true” for those windows that may contain a face or “no” otherwise. The goal is 
to discard windows that may not have faces in it. The classifier is built as a sequence 


(cascade) of (weak) classifiers (Figure 3) in which each consecutive classifier is stronger 


than the previous one. These weak classifiers have been previously trained before the 





face detection phase is executed by using the AdaBoost algorithm [13]. Once the 
windows classified with “yes” have been labeled by the cascade classifier, they may be 


passed to more complex algorithms. 


In recent years, there have been advancements in face detection using deep learning 
methods. Based off of the work from Viola — Jones, there has been success in the 
performance of deep learning face detection algorithms using deep convolutional neural 
networks (CNN), region-based CNN (RCNN), and Faster R-CNN [15]. Most of the recently 
developed methods stem off of the Faster R-CNN and are often able to outperform 
traditional computer vision methods by a significant margin in both accuracy and speed. 
One such method is the Faster R-CNN coupled with region proposal networks (RPN). An 
RPN simultaneously predicts the bounds of an object and objectness scores at each 


position, which are used by Faster R-CNN for detection [16]. 
2.3 General Methods for Bystanders’ Facial Privacy Protection 


Methods currently available to handle bystanders’ facial privacy can fit into two major 
groups: location-dependent methods, which deny third-party devices the opportunity to 
collect data; and obfuscation-dependent methods which prevent bystanders’ facial 
detection and identification. The taxonomy used in this paper to classify the methods to 


protect bystanders’ facial privacy is presented in Figure 4 below. 
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2.3.1 Location-Dependent 


The goal of location-dependent methods is to deny the collection of data at particular 
shared spaces. Implementation of these methods (such as restaurants, casinos, or cafes) 
entails restricting and banning devices’ use through warning signs, confiscating devices 
before entering a shared space, or temporarily disabling user devices in a shared space. 
According to the taxonomy presented in Figure 4, these methods can be further 
classified into two categories, namely (1) banning/confiscating devices; and (2) disabling 


devices. 


In the banning/confiscating devices category, third-party devices are confiscated or 
banned for usage at a shared space. This method has been in use since the end of the 
19th century when the use of cameras was forbidden at private beaches and, for some 
time, at public spaces in the U.S. [17]. As devices cannot be used at the shared space, 


the bystanders’ facial privacy is protected. 


In the disabling devices category, bystanders’ facial privacy is protected because third- 
party devices cannot collect data about the bystanders. Devices can be disabled in 
shared spaces by using three approaches: sensor saturation, broadcasting commands, 
and context-based approaches. In the first approach (sensor saturation), the goal is to 
make sensors of third-party devices sense an input signal that is greater than the 
maximum possible measurable input supported by third-party devices’ sensors (thereby 
making the sensors unusable by saturation). An example in this category includes using 
near-infrared pulsating lights from fixed devices at shared spaces directed at the 
device’s camera lens [18] with the goal of saturating the Charge-Coupled Device (CCD) 
sensor. Facial privacy is preserved because data cannot be collected when the device’s 


sensor saturates. 


In the second approach (broadcasting commands) under the disabling devices category, 
the third-party devices receive some type of command via wireless communication to 


disable temporarily the capture of facial data. An example of this category includes the 


utilization of Bluetooth and infrared protocols to send disabling commands [19 — 20]. In 





the last category (context-based approaches) under location-dependent methods, third- 
party devices perform some type of context recognition to trigger software actions that 
will deny the explicit collection of data by disabling user devices’ sensors at shared 


spaces. 


An example in this category includes the virtual walls approach [21] in which the device 
uses contextual information (such as GPS location data) to trigger software actions that 
can temporarily disable its sensors based on pre-programmed contextual rules. A 
second example in this group is the system developed by Blank et al. [22] in which 
camera-enabled drones are restricted from flying over certain areas through rules 
established in a website and broadcast to the drones. In this case, bystanders’ facial 
privacy is preserved because data cannot be collected by third-party devices when the 


contexts are recognized, and the device’s sensors are disabled. 
2.3.2 Obfuscation-Dependent 


Obfuscation methods attempt to hide the identity of bystanders to avoid their 
identification. These methods can be classified in two groups: (1) bystander-based 


obfuscation; and (2) device-based obfuscation. 


In bystander-based obfuscation, bystanders take actions to avoid their facial 
identification. This might be accomplished by wearing some type of hardware (or 
clothing) that hides or perturbs bystanders’ identifiable features needed to perform 
identification, or by having bystanders perform some type of physical action (for 
example, leaving the shared space, or asking a user to stop using a device) to protect 
their privacy when bystanders become aware of a device’s use in their surroundings that 
might infringe upon their privacy [23]. Examples in this category include the PrivacyVisor 
glasses [14] [24] that hide facial features using near-infrared light or reflective materials, 
and the utilization of wearables to impersonate or to hide facial features to deceive 
facial detection and recognition algorithms [25]. Notification methods that alert 


bystanders to protect their privacy include the use of LEDs on wearables to notify 


bystanders of video or audio being recorded in their surroundings (such as Snap 





10 


spectacles), and the use of short-range radio broadcasts and WiFi-based communication 
protocols to notify bystanders about sensing activity being performed in their proximity 


(e.g., NotiSense [23]). 


In the last group (device-based obfuscation), the software of third-party devices adds 
noise (such as blurring) on collected data to hide bystanders’ facial identifiable features. 
The software at users’ devices might perform obfuscation by default (for example, 
blurring all faces detected in a photo or a video), it might let users add noise to 
obfuscate bystanders selectively (selective obfuscation) [26], or the software on the 
users and bystanders’ devices might access protocols over wireless networks to 
communicate privacy settings such that the software on the user device could 
automatically hide bystanders’ identifiable features based on these privacy settings 
(collaborative obfuscation) [27]. The drawback of device-based obfuscation is that 
bystanders might have no control on protecting their privacy because device-based 


obfuscation methods rely on third-party devices for which bystanders have no control. 
2.4 Design Issues and Performance Evaluation of Current Methods 


Even though solutions to address the issue of bystanders’ facial privacy have been 
proposed in the past (as described in the previous sections), these solutions have issues 
that depend on the type of method and their implementation. Some of these issues that 


affect these solutions are as follows: 


e Usability: In human-computer interaction, usability is described as how easy a 
system can be used by a typical consumer/user to fulfill its objectives. In systems 
to enhance bystanders’ facial privacy usable systems should minimize user 
intervention by the bystander. 

e Power consumption: In any type of battery-powered system, power 


consumption plays a substantial role because devices that deplete their battery 


in a fast manner need to be recharged often. Since many solutions for 





ila 


bystanders’ facial privacy protection involve the utilization of algorithms in 
mobile devices, power consumption is an issue for these systems. 

e Effectiveness: Solutions to protect bystanders’ facial privacy involve components 
and algorithms to identify contexts/faces (to blur or obfuscate them), while 
others involve extra devices or contraptions combined with intelligent 
algorithms. Since these systems make use of artificial intelligence algorithms 
(i.e., classification algorithms) to detect these contexts and/or faces, these 
solutions may involve false detections or misclassifications which hinders the 


effectiveness for the system to work correctly. 


Table 1. Design issues for bystanders’ facial privacy solutions. 














Design Issue Description Rating 
Usability Is the method easy to use? Low, Moderate, High 
Power Consumption Does the method require high Low, Medium, High 
power consumption? 








Effectiveness Is the method effective to Low, Medium, High 


protect bystanders? 














Based on these issues, the methods available for bystanders’ facial privacy are evaluated 
by using the ratings for each category as presented in Table 1. The evaluated methods 


along with their corresponding ratings are described in Table 2 below. 


Table 2. Methods for bystanders’ facial privacy protection 


























Method Category Usability Power | Effectiveness Remarks 
: Location 
BlindSpot (disablin Utilization of InfraRed (IR) light to 
Capture-resistant Ponce High Low Low disable CCD sensors may not be 
sensor useful with IR filters on modern 
environment [18] cameras. 
saturation) 
Disabling devices ; High Low Medium Method requires third-party 
ta Infeaved:[19 Location devices to receive IR commands 
vig inttanedt [22) and software to disable sensors 

















(disabling, 
sensor 


saturation) 
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which not all third-party devices 
may the capability. 








































































Location : , eer) 
Disabling devices Method requires third-party 
(disabling, F : F devices to receive Bluetooth 
via Bluetooth Sanwa 7 Yee ene commands and software to disable 
[20] sensors which not all third-party 
i saturation) devices may have the capability. 
Location Method requires bystanders to 
(disabling ; : setup privacy rules that are 
Virtual Walls [21] 3 Moderate | High Medium accessed in third-party devices. Use 
sensor of sensors in mobile device to 
; determine contexts may consume 
saturation) 
large amounts of power. 
begaon Method requires bystanders to 
Privacy-restricted (disabling, Moderate | Medium Medium setup privacy rules that are 
accessed in third-party devices. 
areas [22] sensor : 
Proposed for unmanned aerial 
saturation) vehicles. 
Location 
World-driven 
(disabling, High High Medium Method does not require 
access control bystanders’ intervention, but device 
[28] ane may not detect contexts correctly. 
saturation) 
aeasianded Method does not require 
Sensor Tricorder (disabling, High High Medium bystanders’ intervention, but device 
[29] pineal may not detect contexts correctly. 
Makes use of QR codes to encode 
saturation) location privacy rules. 
Require machine learning 
Location algorithms to detect sensitive 
(disabling : ; contexts. May not detect contexts 
PlaceAvoider [30] ’ Moderate High Medium correctly. Devices must have 
sensor software to detect contexts. 
; Requires third-party user 
satucation) intervention to check if areas are 
indeed sensitive. - 
scopy doc ale Require third-party devices to 
; based Moderate Low Medium notify bystanders about possible 
NotiSense [23] ' 4 privacy violations and have the 
ioe reMee ss bystander to take action to protect 
based) their facial privacy. 
Obfuscation- 
Use of IR in wearables worn by 
PrivacyVisor [24] based High High Low bystanders to obfuscate facial 
(bystander- features. IR can be blocked using 
| filters. 
PrivacyVisor lil Obfuscation- High how High Use of reflective materials in 





[14] 





based 





wearables used by bystanders to 
corrupt photos taken about them. _ 
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(bystander- 
based) 
Obfuscation- 
Perturbed Use of patterns in glasses’ frames to 
based Medium i iti 
eyeglass frames confuse facial recognition 
(bystander- algorithms. May be prone to 
[25] reidentification. 
based) 
Oencanet Use of IR in wearables worn by 
Invisibility based High low bystanders to obfuscate facial 
features. Need high power and IR 
Giese ies] ieyetener can be blocked using IR filters which 
based) are available for mobile phones. 
Privacy Obhuscatone a technology does not depend 
on the bystander but on the 
Protection in based High low High company collecting photos. 
Google bystander: Company performs obfuscation in 
the cloud after the photos have 
StreetView [32] based) been forwarded from the device 
that captured them. 
Pata On This technology blur faces in photos 
based F : F through a mobile app. Face blurring 
ObscuraCam [26] ee oe mee occurs at the mobile phone and 
pyeeneer depending of the blurring technique 
based) bystanders could be re-identified. 
. Use of protocols between 
atic a bystander and third-party devices 
; based Moderate High Medium to allow/deny blurring based on 
I-pic [27] privacy rules. Face blurring occur at 
eee? the mobile phone and depending of 
based) the blurring technique bystanders 
could be re-identified. 
3 Use of protocols between 
a ge ag a pea and third-party devices 
PrivacyCamera based Moderate High Medium to allow/deny blurring based on 
privacy rules. Face blurring occur at 
[33] isiander: the mobile phone and depending of 
based) the blurring technique bystanders 
could be re-identified. 
Obfuscation- 
Bystanders use visual colored cues 
Respect Bare High Low High to inform capturing device of 
Cameras [34] (bystander- privacy rules. Developed for fixed 
cameras. Face is fully hidden. 
based) 
abticcudie. Use of protocols between 
bystander and third-party devices 
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Chapter 3. System Description 


3.1 FacePET System 


In this section, we describe the Facial Privacy Enhancing Technology (FacePET) system 
developed in conjunction with NSF REU students, Luis Y. Matos Garcia and Jaouad 
Mouloud. The FacePET system is based on the idea that bystanders’ facial privacy should 
be handled by the bystander instead of relying on third-party devices to control 
bystanders’ facial privacy. To this end, we have developed a prototype of a smart 
wearable device that uses visible light to create noise to distort the Haar-like features 
used by face detection algorithms, therefore our wearable allows bystanders to protect 


their privacy. 


We have incorporated a Bluetooth Low Energy (BLE) microcontroller that controls when 
the lights are enabled/disabled based on privacy rules established by the bystander. The 
goal on the utilization of the BLE microcontroller is for the bystander to provide consent 
to third-party devices who may want to take photos of the bystander. Our work is 


similar to the work of Yamada et al. [24] with the following differences: 


e |n Yamada’s work [24] the authors propose the use of near-infrared light to 
saturate the Charged-Coupled Device (CCD) sensor of digital cameras to distort 
the Haar-like features. In contrast, our work uses visible light. The reason to use 
visible light is that newer cameras in smart phones (e.g., Apple’s iPhone 4 and 
newer) and other devices may include an IR filter that blocks the intended noise 
if IR light is used. This makes their device unsuccessful in protecting bystanders’ 
facial privacy. 

e Our system includes a BLE microcontroller for the bystander to control an Access 
Control List (ACL) in which the bystander can setup permissions for third-party 


devices to take photos without the noise (disabling temporally the FacePET 


wearable), hence creating a “smart” wearable. 
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e The development of a wireless protocol over Bluetooth that enables 
communication between the bystander and third-party devices to provide and 


exchange privacy consents. 





FacePET wearable 


Wires 
Bluetooth (( BLE | fen —_———P £-») 








|  BLE-enabled 
. | microcontroller Goggles with 
lad Bystander’s | with power LEDs 
8 mobile phone | supply 





mobile phone = 


Figure 5. FacePET system’s hardware architecture. 


3.2 FacePET System’s Hardware Architecture 


The hardware architecture of the FacePET system (presented in Figure 5) is composed of 


the following components: 


e Goggles with LEDs: The goggles are equipped with LEDs that are turned on/off by 
the microcontroller. To avoid physical discomfort to the bystander when using 
the goggles and the LEDs are turned on, the goggles’ lenses should have a filter 
tuned to the wavelength of the LEDs on the goggles. The LEDs on the goggles are 
connected to the BLE-enabled microcontroller through wires which also provides 
power to them. 

e BLE-enabled microcontroller: This component controls the LEDs on the goggles 
and connects to the bystander’s mobile phone via Bluetooth Low Energy (BLE). 
The microcontroller has its own power supply independent to the one in the 
bystanders’ mobile phone that also provides power to the LEDs. Depending on 
the privacy protocols implemented, the microcontroller may have the software 
that implements the ACL to disable the LEDs, or the ACL may be implemented at 


the bystanders’ mobile phone software. The FacePET wearable is composed of 


the BLE microcontroller and the googles (as shown in Figure 5). 
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Bystanders’ mobile phone: The bystanders’ mobile phone executes software that 
configures the wearable’s microcontroller. In addition to configure the wearable, 
the bystanders’ mobile phone executes software that provide consent to third- 
parties to turn off the LEDs when an authorized third party wishes to take a 
photo with the bystander in it. Depending on the privacy protocols 
implemented, when an authorized third-party wishes to take a photo with the 
bystander, the ACL may be implemented in the bystander’s mobile phone or the 
third-party may communicate directly with the wearable. The bystanders’ 
mobile phone communicates via BLE with the microcontroller and it 
communicates with third-party mobile phones via Bluetooth. In future 
implementations, this communication between smartphones may also be Wi-Fi 
or IP-based communication. 

Third-party (stranger) mobile phone: The third-party (stranger) mobile phone is 
used by a third-party to request consent for photos to be taken of the bystander. 
In our current implementation, these consents are requested via Bluetooth to 
the bystanders’ mobile phone prior to when the third-party can take a photo of 
the bystander. If consent is given by the bystander, when the third-party mobile 
phone takes a photo of the bystander, it communicates with the bystander 


device again to request the LEDs of the goggles to be turned off (if consent has 


been given previously). 
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ZAM! 





Figure 6. The FacePET wearable device. (a) Wiring sketch diagram for FacePET LEDs; (b) Goggles 
with LEDs and BLE microcontroller; (c) FacePET wearable prototype worn by a bystander (the 
person in the photo is Jaouad Mouloud) 


In our current prototype we used safety goggles bought at a local hardware store. We 
placed six LEDs on the goggles as shown in Figure 6(c). Initially we tried IR LEDs, but they 
were discarded when we found that the Apple iPhone 4 and newer versions of the 
iPhone include an IR filter for their rear-facing camera (possibly IR filters will become a 
standard feature in future mobile phones). As a consequence, we tested red, green and 
blue LEDs for our prototype. For the BLE-enabled microcontroller in the prototype, we 
used an Arduino Uno [38] with the Seeed Studio Bluetooth 4.0 Low Energy-BLE Shield 
v2.1 [39] (Figure 6(b)). The Arduino’s power supply used was a battery pack connected 


to the Arduino’s USB-B port. Figure 6(a) shows the wiring sketch diagram for the 
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Arduino board and the LEDs. We used smartphones that support BLE and can run 


Android 6 (or better). 
3.3 FacePET System’s Software Components 


To control the FacePET wearable device and implement the bystanders’ consent 


protocol, we developed the following software: 


e FacePET microcontroller’s software: In the current implementation of the 
FacePET wearable, this component allows the functionality of turning on/off and 
changing the intensity of the goggle’s LEDs (in groups of two LEDs 
independently) and providing a mechanism to control these LEDs from the 
bystanders’ mobile phone via Bluetooth Low Energy (BLE). Since we built the 
wearable with the Arduino Uno and the Seeed Studio BLE Shield, the RBL_nf8001 


and BLE-SDK Arduino libraries were used to create a Generic Attributes (GATT) 


BLE server that is used to receive commands from the bystander’s mobile phone. 
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Figure 7. FacePET’s system mobile app screenshots. (a) Bystanders’ app; (b) Stranger (third-party) 


FacePET bystander’s mobile app: This application provides the bystander a 
controller for the FacePET wearable via BLE to turn on/off and change the 
intensity of the LEDs, it implements the ACL for the FacePET wearable, and it also 
implements a Bluetooth protocol that provides the bystander wearing the 
FacePET wearable device a mechanism to give consent to third-parties to take 
photos. Initially, the FacePET bystanders’ app scans for a FacePET wearable in 
the area and once connected to it, it enables the LEDs in the wearable. The LEDs 
stay powered on until the bystander turns them off, or a third-party FacePET 
(stranger) mobile app with consent requests a photo to be taken. The protocol to 
provide consent is described in section 3.4. Screenshots of this mobile app are 


shown in Figure 7(a). 
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e FacePET third-party (stranger) mobile app: This app provides a third-party 
(stranger) a mechanism to ask for consent to take photos from the bystander via 
Bluetooth. Once consent is given, the app will send a command to the FacePET 
bystander’s mobile app to disable temporarily the FacePET wearable (as 
described in section 3.4). Screenshots of this mobile app are shown in Figure 


7(b). 
3.4 FacePET System’s Consent Protocol 


As a bystander’s surroundings and context may change over time, he/she may not 
notice when somebody may be taking photos of him/her without consent. One of the 


features and contributions of the FacePET system is the communication protocol that 


(fay) ) 


Third-party (stranger) 


mobile phone FacePET 


; wearable 


Turn on | Scan for third-party wrstenee n 

“Discoverability” in app devices aiting for commands 
Waiting for incoming Third-party device found 
messages and MAC saved in ACL 


| Authorization message | 


Bystanders’ Bluetooth 
MAC saved 


Waiting to authenticate 






Send request for 
authentication 
(to take photo) 


Third-party device 
cleared 


Authorization Turn off LEDs 
message/ Take photo 


Photo taken 


Photo taken 
Turn on LEDs 


Figure 8. Sequence diagram for FacePET’s consent protocol. 
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provides a bystander wearing the FacePET device a way to give consent, therefore 
protecting the bystander’s facial privacy and enabling a mechanism to create a list of 


“trusted cameras” for the bystander. 


The protocol (implemented over Bluetooth in our prototype and shown in Figure 8) 
enables the bystander to control an ACL in the FacePET bystander’s mobile app to 
enable/disable the FacePET wearable’s LEDs when a trusted third-party mobile phone 
wants to take photos. Now, we will describe a scenario in which three personas, namely 
Betsy (a bystander using the FacePET system), Trisha (a third-party using the FacePET 


third-party app) and Steve (a third-party, stranger with a camera) interact at a party. 


Initially, Betsy is wearing the FacePET system with the LEDs on. Trisha and Besty are 
friends and trust each other. Trisha asks Betsy if she can take pictures of her during the 
party, either by talking to her or through an Internet messaging app (e.g., WhatsApp). If 


Betsy does not want Trisha to take photos, she simply ignores the message. 


However, if Betsy desires to give consent to Trisha to take photos of her, Betsy replies to 
Trisha by asking her to open the FacePET third-party (stranger) app and to press 


“Discoverability”, then the following steps take place over Bluetooth: 


1. Betsy opens the FacePET bystander’s mobile app and scans for Bluetooth devices 
to get Trisha’s Bluetooth MAC address and device name. 

2. Once Trisha’s device is found via Bluetooth, Betsy authorizes Trisha’s device and 
the bystander’s app saves Trisha’s Bluetooth MAC address and device name ina 
file (Betsy’s app adds Trisha’s device to the ACL). 

3. Betsy’s FacePET bystanders’ app sends a message via Bluetooth to Trisha’s 
FacePET app notifying that her device is cleared to take photos of Betsy. At this 
point Betsy’s FacePET’s app creates a Bluetooth server socket to wait for photo 
requests from Trisha’s FacePET app. 


4. Trisha’s app saves Betsy’s Bluetooth address so it can be used later to request 


Betsy’s FacePET wearable’s LEDs to be turned off (as long both mobile phone 
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devices are in range and Betsy’s FacePET mobile app still has Trisha’s phone 


authorized in the ACL). 


Later in the party when Trisha wants to take a photo of Betsy, the following steps are 


followed: 


1. Trisha opens her FacePET mobile app. She presses the “Take Picture” button 
and selects Betsy’s device from the list. Trisha’s device then sends an 
authentication message to Betsy’s device via Bluetooth. 

2. The authentication message is received by Betsy’s FacePET mobile app. The 
mobile app then checks if the Trisha’s device is authorized in the ACL. If it is, 
then it notifies back to Trisha’s app that her device can take the photo, and it 
sends a message via BLE to Betsy’s FacePET wearable to turn off the device. 
Otherwise, Betsy’s app will ignore the message and the LEDs will stay on. 

3. Trisha takes the photo and then it sends a message back to Betsy’s FacePET’s 


mobile app to turn on the LEDs again. 


During the party, Steve (a stranger with camera) has tried to take photos of Betsy’s face. 
Since he doesn’t have permission from Betsy, all the photos he takes from her will look 


similar to Figure 6(c) thus protecting Betsy’s facial privacy. 


| With the sensors in the bystander’s mobile phone, more complex privacy rules could be 
created to provide consent. For example, we tested a simple modification in which a 
trusted camera can take only a certain number of photos and after the max number of 
photos authorized has been reached for that camera, the FacePET wearable’s LEDs will 


remain powered on. Other contexts may include location, activity or time by modifying 


the FacePET bystander’s app to manage the ACL using context-based privacy rules. 
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Chapter 4. FacePET Evaluation 


4.1 Evaluation Goals 


When creating the FacePET wearable device, we had two goals in mind that we wanted 
to evaluate: usability and effectiveness. For usability, we wanted the interaction 
between the user and the device to be as easy as possible. For the bystander to setup 
and work the device as well as control their preferences in the application of who they 
allow to take their picture should take minimal effort. The same goes for the 
accompanying application for the stranger and their preference control. As for the 
effectiveness of the device, the goal was to observe if the wearable device was effective 
in protecting a bystander’s facial privacy using the FacePET wearable independently of 
the camera being used. The lights around the device are placed in such a way that they 
hide the Haar-like features of the individual’s face well enough to fool face detection 
algorithms. These two goals were the main focuses of the device going forward into its 


evaluation. 
4.2 Methodology 


In order to recruit and collect data from research participants, the necessary 
Institutional Review Board (IRB) application needed to be filled out and approved. Upon 
submission, the application was approved on the date of May 14, 2018 and given the 
approval protocol number 18-108. The initial recruitment of participants was carried out 
by the supervising professor, Dr. Alfredo J. Perez, who emailed the recruitment flyer to 
professors in the Computer Science department. The flyer explained that individuals 
who wanted to take part in the research study were to come to Room 123 in the 


Synovus Center of Commerce and Technology building on the CSU campus. 


Once the participants entered the room, they filled out the informed consent form so 


that they understood what was taking place. Next, they filled out an initial survey about 


the general concept of bystanders' privacy as well as their personal preferences on 
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having their photos taken in certain situations. Then, the participants wore the FacePET 
wearable device and had their photo taken with the device turned on and off. These 
photos were then used as input in a Python script that makes use of the OpenCV face 
detection API [40] which provides an open source implementation of the Viola — Jones 
face detection algorithm. Lastly, the participants filled out a second survey regarding the 
wearable device itself and how they felt about it, concluding their participation. A total 
of 21 participants were surveyed in this study. The results from the study will be 


presented using tables and graphs in the following section. 
4.3 Results 


4.3.1 Bystanders’ Privacy Survey 


The initial bystanders’ privacy survey served as a way to gain information about each 
participant’s knowledge of what bystanders’ privacy is and how it affects them. 
Participants were first asked lead-up questions about if they considered themselves a 
tech savvy person and how often they took pictures and videos. They were also asked 
how much they knew about the issue of bystanders’ privacy and if they found it to be an 
important issue in today’s world. The results to these questions will be discussed later in 
section 4.4. The participants were then asked to imagine themselves being 
photographed in certain situations and to choose the privacy action they would be most 


comfortable with. These results are presented below in Table 3 and in Figure 9. 


Table 3. Participants’ preferred privacy actions regarding various situations. 
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~*—atthegym 

~*~ engaging in a daily outdoor activity (e.g. walking cycling, going 
to market places, etc.) 
in a bar ora nightclub 

*~atthe beach 

~#—at my workplace 

~*~ata place of worship 

—*— using public transportation 

~¢—ata hospital 

—*—in a restaurant 

—*at a private gathering with family or friends (e.g. birthdays, 


weddings, etc.) 


~*—at a public gathering (e.g. exhibitions, concerts, movies, etc.) 


Figure 9. Chart of participants’ preferred privacy actions regarding various situations. 


After giving their privacy actions for certain situations, the participants were then asked 


how some given factors would affect their comfort level when being photographed. This 


was regardless of any specific situation. The results for this part of the survey are shown 


below in Table 4 and in Figure 10. A final question put the participants in a 


photographer’s position and asked if they would like to respect the privacy preferences 


of the people around them. These results will be discussed later on as well. 


Table 4. Participants’ comfort levels regarding various factors. 











Choice A) | will Choice 
Choice B) | will Choice D) I will | Choice E) | will 
feel much C) I will 
COORONG WHC. feel a bit more feel a little less | feel much less 
more feel the 
comfortable comfortable comfortable 
comfortable same 
The photographer is a ‘ 
professional photographer 
13 4 4 0 0 
(e.g. wedding photographer, 
journalist, artist, etc.) 
il 























28 














The photograph will be i =o 
limited to personal use by 5 3} 7 4 2 
the photographer 
I There are minor children in a wail | 
your vicinity who might also 0 0 9 9 3 
be photographed 
ia: 4 








The photograph may be 
published online and | am 
notified afterwards (e.g. 


social networks) 








The photograph may be 
posted in a forum with 
restricted membership (e.g. 4 4 8 3 2 


company/university mailing 





list) 





The photographer is an 


acquaintance 





The photographer is a 





stranger 





| am photographed while | 


am with strangers 








1! am photographed while | a ; | 


am with acquaintances 








The photograph may be 


published online without my 





knowledge (e.g. social 


networks) 























29 


Participants’ Comfort Levels 
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Figure 10. Chart of participants’ comfort levels regarding various factors. 


4.3.2 Wearing the FacePET System 


After participants were finished with the bystanders’ privacy survey, they wore the 
FacePET system and were explained to in detail how the Bystander and Stranger 
applications worked. Each individual was photographed using the rear-facing camera of 
an Apple iPhone 7 mobile phone with the device’s lights turned on and off, and those 
photos were used as input in the OpenCV face detection script to show how the device 
could effectively hide the Haar-like features used in the face detection algorithm. Out of 
the 21 tests done when taking pictures with the device’s lights on, 6 of the participants’ 
faces were still partially or completely detected by OpenCV. This gives a success 


percentage of around 71%. 


A handful of the participants also took pictures using their own mobile phones so that 
comparisons could be made for how effective the device worked regardless of the 


different cameras. For the entire experiment, green LEDs were used for FacePET. The 


results for face detections using different mobile phones are presented in Table 5 and 
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are a combined effort from the experiment described in this section and the 


experiments of Luis Y. Matos Garcia and Jaouad Mouloud. 


Table 5. Results from FacePET facial privacy protection with different rear-facing cameras 
and OpenCV face detection library. FacePET wearable with green LEDs. 













































































Mobile phone Basic camera features i detected? | 
(Rear camera; Front Camera; IR filter) 
Apple iPhone 6 Plus R: 8 MP; F: 1.2 MP; IR: Yes No 
Apple iPhone 7 Plus R: 12 MP; F: 7 MP; IR: Yes No 
Apple iPhone 8 R: 12 MP; F: 7 MP; IR: Yes No 
Apple iPhone 8 Plus R: 12 MP + 12MP (dual cameras); F: 7 MP; IR: Yes No ! 
Apple iPhone X R: 12 MP; F: 7 MP; IR: Yes No na 
Samsung Galaxy S7 R: 12 MP; F: 5 MP; IR: No Yes 
Samsung Galaxy S7 Edge R: 12 MP; F: 5 MP; IR: No No 
| 
Samsung Galaxy S8 R: 12 MP; F: 8 MP; IR: No No 
Samsung Galaxy S9 R: 12 MP; F: 8 MP; IR: No No 
Samsung Galaxy S9 Plus R: 12 MP + 12MP (dual cameras); F: 8 MP; IR: No No ao 
Samsung Note 7 R: 12 MP; F: 5 MP; IR: No | No | 
Samsung Note 8 R: 12 MP + 12MP (dual cameras); F: 8 MP; IR: No No 
Asus ZenFone 3 Max R: 16 MP; F: 5 MP; IR: No No- é 
Asus ZenFone 4 R: 12 MP + 8MP (dual cameras); F: 8 MP; IR: No No 
OnePlus 6 R: 16 MP + 8MP (dual cameras); F: 16 MP; IR: No Yes 
Motorola Moto G (2"4 Gen) R: 8 MP; F: 2 MP; IR: No i gillian 
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4.3.3 Wearable Device Survey 


The final part of the study had the participants complete a wearable device survey 
about the FacePET system. It questioned the participants about the usability of the 
device, if the device was something that they would use daily and if not, would they use 
a similar version of the device. If a participant decided they would not wear a similar 
version of the device, they could give their reasons as to why that is. The next question 
asked them what they think the reactions of people would be when seeing them 
wearing the device. They were also asked that if wearables that concealed users’ 
identities became available, will they allow smart glasses to become more popular. 
Finally, the survey concluded by asking participants if there were any improvements to 
the FacePET system that they would recommend. Results to these questions will also be 


discussed in the next section. 
4.4 Discussion of Results 
4.4.1 Bystanders’ Privacy Survey Discussion 


The first set of questions in the bystanders’ privacy survey were able to give insight into 
participants’ practices and knowledge with regards to technology and bystanders’ 
privacy. Out of the 21 total participants, 19 of them considered themselves to be tech 
savvy while 2 of them thought not so much. When asked how often they took pictures, 
videos, etc., 3 participants said very often, 4 said pretty often, 4 said often, 8 said not so 
often, and 2 said very little. The participants were then asked specifically about the issue 
of bystanders’ privacy and how much they knew of it. Surprisingly, most of them did not 
know much about the issue if anything at all with 2 saying they knew a lot about it, 8 
said they knew enough, 8 did not know much, and 3 participants did not even know 
what it was. In today’s world, this issue is more evident than it has ever been, yet most 
people still do not know it exists. With that aside, most of the participants were in 


agreement that it is an important issue in today’s world with 18 having said it was, and 3 


saying it was not. 
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Moving on to the preferred privacy actions chosen by the participants when in certain 
situations presented in Table 3 and Figure 9, most of them preferred to either make 
their decision about the photo after seeing it or they do not wish to be captured in any 
photo in places such as the gym or in a hospital. In other situations, such as in a bar or 
nightclub or at a restaurant, most of the participants preferred to make a decision about 
the photo after seeing it above the other preferences. When at private or public 
gatherings, the participants are more open to having any photo taken of them, or if a 
photo is taken then they would want a copy of it. This is understandable since at private 
gatherings, an individual is surrounded by trusted family and friends, while at public 
gatherings, such as exhibitions and concerts, almost anyone around will have their 


phone out taking photos and videos of the event. 


Looking at Table 4 and Figure 10, the participants were presented with a new set of 
questions about how comfortable they would be with different factors affecting them 
when being photographed. In the presence of a professional photographer or if the 
photographer was an acquaintance, a majority of the participants chose that they would 
feel a bit more comfortable if not much more comfortable with having their photo 
taken. If the factor is that there are minor children in the vicinity who may also be 
photographed, the photographer is a stranger, or the participant is photographed with 
strangers, the comfort levels of the participants mainly decreased with them feeling 
either the same, a bit less comfortable, or much less comfortable. Having minor children 
captured in photos can be a very sensitive issue depending on varying factors, and when 
the photographer is a stranger, or an individual is being photographed with strangers, 
other privacy issues come into play since other people who are not trusted are handling 


the captured images. 
4.4.2 FacePET System Experiment Discussion 


It was stated before that of the 21 consecutive pictures taken of the participants’ faces, 


6 of them were still detected by OpenCV. This is good, but it calls into what factors 


might be causing almost a third of the faces to be detected. During some of the studies, 
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it was noticed that the glasses seemed a bit big on some of the participants who had 
thinner or smaller facial structures. This caused more of the Haar-like features to still be 
seen through the lenses themselves rather than being blocked in the areas where the 
LEDs were. There was also an issue with the lighting of the area, where some of the light 
reflections were mistakenly caught by OpenCV as maybe a glimmer of the eye. Different 


lighting environments could have a significant effect on the effectiveness of the device. 


Analyzing the results from Table 5, it can be seen that OpenCV was able to detect faces 
only in photos taken with the Samsung Galaxy S7 and the OnePlus 6 mobile phones (2 
out of 16 devices tested). This shows that using green LEDs for FacePET is effective in 
protecting a bystander’s facial privacy. Before this experiment, it could be assumed that 
nicer mobile phone cameras would make it difficult for FacePET to work properly since 
more detail could be captured. That is certainly not the case seeing as the Apple iPhone 
8, iPhone X, and the Samsung Galaxy S9 all came out within the past few years or so and 


OpenCV still could not detect the faces of individuals. 


Regarding the actual uses of the applications for FacePET (the Stranger app in 
particular), the functionality worked smoothly until the stranger wanted to take a 
picture. Even when having permission from the bystander to take their picture, the 
camera would not open up at all on occasion. This could be due to communication 
errors between the Stranger and Bystander applications, or it could be a software issue 


which can be fixed. 
4.4.3 Wearable Device Survey Discussion 


Having had a chance to see how the FacePET system worked, 17 of the 21 participants 
found the device easy to understand and use, while only 4 found it more difficult. This 
means that the layout and functionality of the applications was made easy enough for 
the majority of users to pick up in a small amount of time. When asked if the device was 
something the participants would use daily, 9 said yes while the other 12 said no. Out of 


those 12, they were asked if they would use a version similar to FacePET with 7 saying 


yes and 5 saying no. Even though the original system is not something most of the 
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participants would use, a majority of them would use a similar version. For those 
participants who said no to using a similar version of the device, they were asked for 


reasons as to why with some of the reasons including: 


e The current model is too big and draws attention 

e The model is not stylish and can obstruct vision 

e Select participants do not really take pictures or engage in the media market in 
such a manner 


e Select participants would use a different form of the device, such as a watch 


Most of the concerns or reasons surrounding participants not wanting to use the device 
seem to be because of the devices form factor. Some of the participants who had 
thinner/smaller facial features found the device sliding down their face, or due to the 
surface area of the device’s lenses compared to some users’ faces, most of the 


identifying facial features could still be picked up by OpenCV as stated previously. 


When the participants were asked how people would react when seeing them wearing 


the device, a variety of responses were given such as: 


e Person laughs and says, “Stupid glasses.” 

e People would stare a lot 

e People would be confused at first or creeped out 

e People would ask why the user was wearing such a device 


e The device would only invite more people to take pictures of it 


It seems there would be plenty of confusion around the purpose of the device and why 
anyone would wear it in its current state. Despite the possible reactions to wearing such 
a device, a majority of the participants did agree that if wearables that conceal users’ 


identities became available, it would allow smart glasses to become more popular with 


17 saying yes, 3 feeling indifferent, and only 1 saying no. 
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To gather some suggestions as to how to improve FacePET, the participants were asked 
to provide any that they would recommend. Some of the improvements that were 


repeated among most of the responses included: 


e Asmaller size of the wearable glasses 
e More LEDs to cover more features, or make them less noticeable 
e Make the device more fashionable/stylish 


e Fix the wiring 


The consensus appears to be that FacePET does not match up with the form factor of 
regular glasses currently available. In order for more people to like wearing the device, 
they need to look more closely to the types of glasses worn in today’s world. This is not 


to say that some people would not like the current form of the device but changing the 


style would improve its chances of being popular among consumers. 
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Chapter 5. Conclusion and Future Work 


In this thesis, we have explored deeper into the growing issue surrounding bystanders’ 
privacy by understanding the various algorithms used in face detection as well as 
evaluating current privacy solutions implemented by researchers over the past years. 
We were also presented with a description and implementation of the FacePET system 
which enables the bystander to hide the Haar-like features used by facial detection 
algorithms by using visible light (green LEDs). Lastly, we analyzed and discussed the 
results of a study carried out to gain an understanding of individuals’ privacy 
preferences, and to evaluate the FacePET system’s usability and effectiveness when 
used by those individuals. Thanks to this study, we were able to conclude that the 
majority of the individuals who partook would be willing to wear FacePET, or a similar 
device, daily for their facial privacy protection, and that if there is an availability of 


wearables that can conceal users’ identities, smart glasses could become more popular. 


There is plenty of work to do in the future when it comes to the FacePET system. Plans 
to improve the system include optimizing its power consumption, changing its form 


factor in later iterations, and the development of context-based rules that may allow 


the bystander to setup privacy rules based on location, time and/or activity recognition. 
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